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EVALUATION 


The  requirement  for  producing  reliable,  low  cost,  quality 
software,  as  expressed  in  such  documents  as  the  Findings  and 
Recommendations  of  the  Joint  Logistics  Commanders  Software  Relia- 
bility Work  Group  (Nov  1975)  and  restated  in  various  conferences 
and  symposium  sponsored  by  the  Department  of  Defense  and  industry, 
has  resulted  in  the  development  of  new  tools  and  techniques,  such' 
as  software  reliability  and  error  prediction  models,  and  in  inves- 
tigations into  the  types  and  causes  of  software  errors,  in  order 
to  find  ways  of  insuring  that  all  future  software  produced  is 
reliable.  However,  much  of  the  research  in  model  development  and 
in  software  error  analysis  has  been  severely  hampered  by  the  lack 
of  sufficient  software  error  data  from  a variety  of  different 
software  projects,  so  that  statistically  valid  conclusions  can 
be  drawn  and  model  predictions  validated. 

This  effort  was  initiated  in  response  to  the  need  for  soft- 
ware error  data,  and  fits  into  the  goals  of  RADC  TPO  No.  5,  Soft- 
ware Cost  Reduction  (formally  RADC  TPO  No.  11,  Software  Sciences 
Technology),  in  particular  the  area  of  Software  Quality  (Software 
Data).  The  report  focuses  on  results  from  the  collection,  cate- 
gorizing, and  analysis  of  over  6000  software  errors  extracted 
from  the  test  and  integration  phase  of  a large  DoD  real-time, 
ground-based  development  project.  The  importance  of  obtaining 
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this  data  is  that  it  can  be  used  to  directly  support  current 
software  error  prediction  model  development,  and  can  also  be 
analyzed  to  discover  any  discernible  patterns  in  the  types  and 
categories  of  errors  as  functions  of  different  software  character- 
istics. In  addition,  the  results  of  analysis  on  this  data  can 
be  compared  with  results  of  similar  analysis  on  software  data 
from  both  real-time  and  non-real-time  projects,  in  order  to  fur- 
ther understand  how  software  errors  are  introduced  and  how  they 
can  be  eliminated  or  controlled.  Finally,  this  data  will  be  used, 
along  with  software  error  data  extracted  from  other  real-time 
ground-based  DoD  software  development  projects,  as  a means  of 
establishing  a baseline  for  real-time  ground-based  software  pro- 
jects in  terms  of  the  types  and  number  of  errors,  which  eventu- 
ally will  lead  to  better  methods  for  controlling  future  real- 
time ground-based  software  projects. 

ALAN  N.  SUKERT,  Captain,  USAF 
Project  Engineer 
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Section  1 
INTRODUCTION 

1.1  SCOPE 

This  document  is  the  final  technical  report  under  RADC  Contract 
No.  F30602- 76-C-0161,  Software  Data  Acquisition  (SDA) . This  nine- 
month  study  focused  on  more  than  6700  software  problem  reports 
for  the  period  from  1 March  1974  through  1 March  1975,  which  was 
the  Test  and  Integration  (T§I)  phase  of  the  software  development. 
Each  problem  was  analyzed,  either  manually  or  by  computer,  as  to 
(1)  the  type  of  error  reported,  (2)  the  point  at  which  the  error 
was  introduced  into  the  development  cycle,  and  (3)  the  corrective 
measure  taken. 

The  report  is  organized  into  five  sections  and  an  appendix.  Sec- 
tion 1 discusses  the  scope  and  objectives  of  the  study. 

Section  2 presents  background  information  about  the  project  that 
produced  the  data  studied.  The  discussion  centers  primarily  around 
a description  of  the  software,  its  development,  the  computer  sys- 
tems used  in  development,  and  types  of  data  used  in  the  study. 

Section  3 describes  the  data  analyzed,  results  from  analysis  of 
the  data,  the  procedures  employed  in  the  analysis  process,  a dis- 
cussion of  the  rationale  involved  in  the  interpretation  of  the 
supplied  error  categories,  and  a summary  of  the  new  error  catego- 
ries defined  by  the  study. 

Section  4 is  a limited  statistical  analysis  of  the  data  acquired. 

Section  5 presents  major  conclusions  and  recommendations:  specifi- 
cally, pertinent  observations,  the  nature  of  problems  encountered 
during  the  study,  and  an  evaluation  of  the  data  used  and  acquired. 

Appendix  A is  a detailed  list  of  the  SDA  error  categories. 

1.2  OBJECTIVES 

The  objectives  of  the  SDA  study  were  to: 

1.  Extract  software  error  data  from  a large,  ground-based, 
real-time  data  processing  system. 

2.  Establish  a software  error  data  base  in  support  of  research 
in  software  reliability  modeling. 

3.  Determine  from  the  software  error  data  acquired,  and  using 
the  error  classifications  supplied,  the  types  of  errors 
experienced  during  the  development  of  the  software. 
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Determine  in  which  phase  of  the  software  development  cycle 
each  error  was  introduced  into  the  system,  and  identify 
the  type  of  correction  applied  to  each  error. 
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BACKGROUND 


2.1  PROJECT  INFORMATION 

The  data  utilized  for  analysis  in  this  study  was  generated  during 
a ballistic  missile  project  designed  primarily  to  respond  to  at- 
tacks or  the  threat  of  attacks  of  Intercontinental  Ballistic  Mis- 
siles (ICBMs).  The  development  of  a large,  real-time  multipro- 
cessor data  processing  system  brought  about  some  unique  situations 
requiring  the  development  of  new  and  sophisticated  algorithms  and 
testing  programs,  and  the  extensive  use  of  simulation.  The  entire 
software  development  effort  was  directed  toward  meeting  the  spe- 
cific needs  of  a real-time,  high-throughput,  reliable  computing 
system. 

2.2  SOFTWARE  DESCRIPTION 

Some  of  the  applications  of  the  data  processing  system  consisted 
of  radar  surveillance,  tracking,  target  classification,  radar 
management  and  testing,  intersite  communication,  and  command  and 
control  display  functions.  Because  the  nature  of  the  system  de-  i 

manded  high  availability,  the  development  of  a maintenance  system 
featuring  rapid  recovery  and  quick  fault  isolation  and  repair  was  | 

required.  The  size  and  complexity  of  the  system  compounded  the  ) 

software  development  problems,  imposing  the  need  for  a system  ' 

exerciser  to  verify  as  much  of  the  system  as  practicable. 

2.3  SOFTWARE  DEVELOPMENT  PROCEDURES 

A major  requirement  during  development  of  the  software  was  a test 
bed  that  accurately  reproduced  the  software  environment,  and  a sys- 
tem of  support  functions  designed  to  operate  on  general-purpose 
computers . 

2.3.1  Requirements  Generation 

A systems  engineering  organization  defined,  established,  negotia- 
ted, documented,  and  rigidly  controlled  system  requirements. 

Changes  to  the  requirements  were  made  as  a result  of  detailed  soft- 
ware design  by  the  development  organizations,  system  test  program 
data,  system  evaluation  efforts,  and  detailed  review  by  the  customer. 

2.3.2  Design 

The  design  phase  consisted  of  two  efforts,  process  design  and  pro- 
gram design.  Process  design  was  the  definition  of  the  system  re- 
quirements translated  into  software  architecture,  global  data  struc- 
tures, tasks,  task  priorities,  and  task  timing  requirements.  A 
task  was  defined  as  a single  unit  or  program. 
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The  process  design  activity  was  complemented  by  program  design, 
which  involved  defining  internal  data  bases  and  developing  algo- 
rithms and  control  structures  for  individual  tasks.  This  combined 
activity  led  to  a detailed  software  specification,  including  spe- 
cific mathematical  equations.  The  design  was  dedicated  to  support 
early  development  of  a system  to  which  greater  capability  could  be 
added  gradually.  Emphasis  was  placed  on  modular  design  to  ease 
system  growth. 

Size  and  execution  time  for  individual  programs  were  two  major 
parameters  that  were  controlled  and  tracked  on  a monthly  basis. 
Design  reviews  were  held  frequently  and  proved  very  effective  in 
planning  for  controlled  and  systematic  changes  and  refinements  to 
the  system. 

2.3.3  Coding  and  Unit  Testing 

During  this  phase  of  software  development,  the  code  was  written  and 
compiled  using  an  IBM  System/360  Model  65  computer.  Programs  were 
written  in  CENTRAN,  an  extensible  intermediate-level  language  re- 
sembling a subset  of  Programming  Language  1 (PL/1).  It  provided 
many  of  the  advantages  of  high-level  languages,  but  could  be  inter- 
spersed with  assembly  language  and  system  macros  when  necessary. 

To  facilitate  preparation  and  testing,  a linkage  editor,  simulator, 
and  disk  library  system  were  developed.  Unit  testing  utilized 
the  simulator  and  drivers  and  was  run  on  the  IBM  System/370. 

2.3.4  Process  and  Functional  Testing 

Tasks  were  blocked  into  processes  and  tested  by  process  integra- 
tion teams  using  larger  drivers  and  system  exercisers.  As  test- 
ing progressed,  processes  were,  in  turn,  blocked  into  functions 
for  more  complex  system  testing. 

2.3.5  System  Integration 

When  testing  achieved  a predefined  level  of  capability,  the  soft- 
ware was  run  on  the  full  complement  of  hardware  using  the  system 
exerciser. 

2.3.6  Evaluation 

Evaluation  played  an  important  role  throughout  the  entire  develop- 
ment cycle.  Evaluation  was  primarily  an  analytical  activity 
which,  because  of  the  complexity  of  the  system,  relied  heavily  on 
simulation.  Also,  because  there  was  a practical  limit  to  the 
level  of  detail  in  which  the  various  weapons  system  functions 
could  be  modeled,  more  detailed  simulations  of  the  particularly 
critical  functions  were  added.  By  employing  simulations  in  con- 
cert, considerable  insight  was  gained  into  detailed  system  oper- 
ation. A feedback  mechanism  in  the  form  of  problem  reports  re- 
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suited  in  frequent  changes  and  refinements  to  the  software,  and 
a constant  updating  of  the  evaluation  simulation  provided  for  a 
more  accurate  representation  of  the  tactical  operation. 

2.4  COMPUTER  SYSTEMS  USED 

2.4.1  Central  Logic  and  Control  (CLC)  System 

The  Central  Logic  and  Control  (CLC)  System  represented  the  appli- 
cation of  the  multiprocessing  concept  to  a large-scale  computing 
system.  A modular  design  was  employed  in  which  as  many  as  ten 
processors  and  two  Input/Output  Controllers  (lOCs)  shared  as  many 
as  32  memory  racks.  Under  software  control  the  CLC  could  be  con- 
figured to  two  separate  partitions  of  arbitrary  size,  each  capable 
of  operating  as  an  independent  computing  system,  and  complete  re- 
configuration could  be  accomplished  in  less  than  one  second.  Ap- 
plication software  executed  on  the  larger  partition,  and  the  ex-~ 
ercise  drivers  and  support  activities  executed  on  the  smaller. 

A single  processor  can  throughput  about  1.5  million  instructions 
per  second  by  means  of  instruction  overlap  and  high-speed  arith- 
metic algorithms.  Since  processors  do  not  communicate  directly 
with  peripherals,  processing  and  input/output  on  the  CLC  occurred 
simultaneously. 

2.4.2  IBM  System  370/165 

The  System  370/165  is  an  information  processing  system  designed 
for  very  high-speed,  large-scale  scientific  and  business  appli- 
cations. The  basic  Central  Processing  Unit  (CPU)  cycle  time  is  .08 
microseconds,  with  a storage  cycle  time  of  2 microseconds.  Approx- 
imately 1.4  million  instructions,  on  the  average,  can  be  processed 
in  one  second.  Contributing  significantly  to  the  speed  and  power 
are  the  main  storage  capacities,  which  range  from  512K  to  3072K, 
and  a high-speed  buffer  storage  that  sharply  reduces  the  time  re- 
quired to  fetch  the  currently  used  sections  of  main  storage.  Speed 
is  further  increased  through  the  use  of  multiple  storage  elements. 
Reliability  and  availability  are  enhanced  through  the  use  of  in- 
struction retry  and  main  storage  error  checking  and  correction. 

2.5  TYPE  AND  EXTENT  OF  DATA  AVAILABLE 

The  data  utilized  for  this  study  was  extracted  from  a data  base 
of  more  than  17,000  problem  reports.  In  accordance  with  paragraph 
4.1.1  of  the  Statement  of  Work,  only  those  reports  written  between 
1 March  1974  and  1 March  1975  (a  total  approximating  6700  reports) 
were  used  in  the  error  data  analysis.  These  problem  reports, 
which  included  hardware  problems,  came  from  various  areas  of  the 
overall  project. 
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2.5.1  Tactical  Software  Errors 

Problem  reports  in  this  category  were  written  against  the  three 
tactical  processes  plus  the  system  exercisers  and  the  global  data 
sets.  There  were  approximately  4320  problem  reports  in  this  area. 


2.5.2 


)ort  Software  Errors 


Problem  reports  in  this  area  included  all  except  those  written 
against  (1)  tactical  software  items,  (2)  hardware  items,  (3)  re- 
ports written  to  identify  suggested  and  implemented  improvements, 
and  (4)  those  reports  classified,  after  analysis,  not  to  be  errors. 
There  were  approximately  1000  problem  reports  in  this  category. 

2.5.3  Hardware  Errors 

Problem  reports  were  written  against  all  facets  of  the  hardware, 
from  burned-out  lightbulbs  to  sophisticated  electronic  design 
errors.  There  were  246  hardware  reports  generated  during  the 
Test  and  Integration  phase. 

2.5.4  Improvements 

Approximately  190  problem  reports  were  written  to  identify  areas 
of  improvement.  Some  of  these  improvements  were  implemented,  but 
the  majority  were  deferred  to  later  periods  when  time  and  funding 
would  be  available. 


2.5.5  Non-Errors 

This  group  of  problem  reports  accounted  for  a significant  number 
(960)  of  the  reports  analyzed  and  can  be  divided  into  three  cate- 
gories. The  largest  number  (709  reports)  consisted  of  duplicates 
of  other  reports.  The  remaining  251  problem  reports  were  con- 
sidered legitimate  non-errors  in  the  sense  that  the  situations 
described  in  the  reports  were  not  in  error  with  the  requirements; 
or  the  problem  was  one  that  existed  only  in  the  simulation  environ- 
ment and  a correction  was  provided  simply  as  an  "accommodation" 
type  of  correction  and  subsequently  removed  when  testing  took 
place  on  the  full  complement  of  hardware.  (These  251  problem  re- 
ports involved  only  those  identified  during  the  manual  analysis 
effort;  many  more  had  already  been  eliminated  during  the  automatic 
analysis  period.) 
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Section  3 
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DATA  ACQUISITION 


3.1  DATA  DESCRIPTION 


3.1.1  Source  Data 

The  data  base  records  of  the  problem  reports  consisted  of  242 
fields,  of  which  only  20  were  used  in  the  identification  and  anal- 
ysis of  the  problem  reports.  Certain  fields  were  used  to  identify 
those  problem  reports  that  were  to  be  used  for  the  study;  other 
fields  were  used  in  the  automatic  and  manual  analyses  of  the  prob- 
lem reports  to  determine  data  such  as  date  of  correction,  type  of 
correction,  phase,  type  of  error,  etc. 

Figure  1 is  an  example  of  the  printed  data  base  record  listing 
those  fields  that  were  pertinent  to  the  SDA  study.  Explanatory 
notes  on  the  page  following  Figure  1 describe  each  applicable 
column  heading  shown  in  the  figure. 

The  Product  Identifier  (PIDENT) , or  program  name,  incorporates  a 
number  of  unique  features.  Figure  2 is  a representative  example 
of  a PIDENT  breakdown;  Table  1 lists  and  describes  the  alphabetic 
characters  used.  The  PIDENT  type  of  program  naming  convention 
facilitates  the  identification  of  the  area  and  function  to  which 
the  program  belongs. 

3.1.2  SDA  Data 

The  data  acquired  for  this  study  was  of  two  types:  data  related 
to  software  errors  and  data  related  to  the  software  development 
process . 

Error- Related  Data:  The  data  gathered  for  this  portion  of  the 
study  dealt  with  software  errors  and  related  statistical  informa- 
tion. Software  errors  were  controlled  and  tracked  by  using  an 
identifying  number  called  a Master  Problem  Report  (MPR)  number, 
and  associated  with  a module  by  way  of  a PIDENT  name.  The  date 
the  error  was  discovered  and  the  date  it  was  corrected  were  main- 
tained as  part  of  the  error- related  data,  along  with  descriptions 
of  the  error  and  its  solution.  The  error  type,  the  means  used  to 
correct  the  error,  and  the  point  in  the  software  development  cycle 
at  which  the  error  was  made  were  items  determined  through  an  anal- 
ysis of  the  source  problem  report  and  stored  in  the  SDA  master 
problem  record. 

Development  Process  Data:  Data  related  to  the  software  development 
process  was  of  the  following  types: 

1.  Computer  Usage  - This  data  represents  the  amount  of  CLC 
CPU  time  utilized  each  month  during  the  T§I  period. 
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NOTES  TO  FIGURE  1 


Column  Heading  Description 


TYP 

MPR  NUMBER 
PIDENT 

MAJOR  STATUS 
MINOR  STATUS 
DATE  WRITTEN 
DATE  CORRECTED 
DATE  SOURCE 
DATE  DOCUMENT 
TESTID 
PUP  ACT 
PUP  SCH 
SITE  ACT 
PCH  DATE 
PCH  SCH 
PCH  TSTD 
DATE  LOG 
DATE  STAT 


DATE  END  TST 
DATE  CR  REC 


DATE  OF  CHANGE 
PCH  NUMBER 
DESC 
SOLN 


Type  of  solution 

Problem  report  number 

Program  name  (Product  Identifier) 

Major  status  code  associated  with  the  problem 

Minor  status  code  associated  with  the  problem 

Date  problem  report  was  created 

Date  problem  solution  was  submitted 

Date  source  code  delivered  from  development 

Date  document  correction  was  delivered 

Test  identification 

Date  patch  actually  put  on  PUP  tape 

Date  patch  scheduled  to  be  put  on  PUP  tape 

Date  patch  actually  sent  to  site 

Date  patch  status  last  changed 

Date  patch  scheduled  for  testing 

Date  patch  finished  testing 

Date  problem  report  logged  into  SAS  system 

Date  of  last  status  change 

Date  end  of  source  testing 

Date  correction  received  by  CSCM 

Date  this  SAS  record  was  last  changed 

Patch  identification  number 

Problem  description 

Problem  solution  description 


TABLE  1.  PIDENT  FUNCTIONAL  BREAKDOWN 

B = BMDC 

S = Logic  Simulation  Facility 
E = System  Exerciser 
C = CLC  Installation  and  Support  Software 
0 = Operating  System 

A = M§D  Buffer  Programs  and  CLC  Monitor  Support 
B = Library 

C = Configuration  Control 
D = Debug 

E = Error  Control,  Interrupt  Handler 
H = Hardware  Test  Scheduler,  Normal  Path  Diagnostics 
1=1/0  Manager 

K = Debugging  Aids  for  Real  Time 
L = Loader 

M = CCDSS  Management,  Man/Machine 
0 = OS  Control 
P = Communicators 

R = RSS  Management,  Overlay  Manager 
S = Scheduler,  Main  Control 
U = Utilities 

X = (functional  level  designation  not  appropriate) 

T = Installation  and  Test  Software  Support  (ITSS)  Facility 
D = DPS  Management  Control 
R = Reporting 
E = System  Exerciser 

G = MSR  and  PAR  Exerciser  Process  Common  Function 
D = Drivers 

G = Global 

X = Routines,  subroutines,  sources  or  data  sets  used  in  more 
than  one  facility  or  process 

X ■ (functional  level  designation  not  appropriate) 

L = System  Test  Tapes 

M = Missile  Site  Radar  (MSR)  Software 

W = MSR  Weapons  Process 


▲ 
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TABLE  1.  PIDENT  FUNCTIONAL  BREAKDOWN  (Continued) 


C = Process  Coordinator 
D = Data  Gathering 
E = Data  Reduction 
F = Interceptor  Response 

G = SPRINT  and  SPARTAN  Guidance,  MDP,  and  Launch 
Area  Control 

L = Tactical  Display  Area 
M = MSR  Site  Manager 
R = Radar  Management 
S = Target  Selection 
T = Test  Coordinator 

V = 360  Driver 

W = Process  Design 

X = (functional  level  designation  not  appropriate) 

Y = Launch  Area  Test 
Z = MSR  Tests 

P = Perimeter  Acquisition  Radar  (PAR)  Software 

I = PAR  Installation  Process 

T = Receiver  Tests  - 2nd  Interval 

M = Independent  Radar  Test  Monitor  (RTM)  and  PAR  Weapons 
(PW)  PIDENTs 

G = Global  Data  Sets 

L = Local  Data  Sets 

P = Process  Coordinator 

R ■ Class  B Radar  Test 

T = PAR  Test  Process  and  RTM  Subprocess  of  PW 
G = Global  Data  Sets 
J = Test  and  Integration 
L = Local  Data  Sets 
P * Coordinator  and  Control 
R * Class  A,  Class  B,  or  Class  C Radar  Tests 
X = (functional  level  designation  not  appropriate) 
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TABLE  1.  PIDENT  FUNCTIONAL  BREAKDOWN  (Concluded) 


ED 


W = PAR  Weapons  Process 

C = Tactical  Communicators/Intrasite  i 

D = Data  Gathering 
G = Global  Data  Sets 
I = PAR  Site  Manager/Intersite 
K = Known  Object  Management 
P = Process  Coordinator 
R = Radar  Manager 
S * Target  Selection 
T » Tracking 

X * (functional  level  designation  not  appropriate) 

R ■ PAR  Trainer  Controller  Program 

T » Training  Task  Initialization 
S » Systems  Engineering 
T ■ Standard  Test  Software 

E ■ 360  Facilities  Standard  Test  Process 
P » Tactical  Operating  System  Cycler 


# 


2.  Statement  Type  and  Rate  - This  data  identifies  the  pto- 
gramming  language  used  in  writing  the  software  and  the 
rate  at  which  an  "average"  statement  in  an  "average" 
statement  mix  was  processed  by  the  CPU. 

3.  Test  Run  Data  - This  data  describes  the  number  of  differ- 
ent test  scenarios  used,  the  number  of  times  all  the  dif- 
ferent tests  were  run,  and  the  percentage  of  tests  that 
ran  to  completion. 

4.  PIDENT  List  - This  data  identifies  all  modules  that  were 
part  of  the  software  system  during  the  T§I  phase.  It 
lists  each  program,  the  size  of  each  in  CENTRAN  state- 
ments, the  language  in  which  each  was  written,  and  the 
mode  of  construction  used  in  development. 

3.2  SPA  DATA  ACQUISITION  PROCEDURES 

The  computer  run  logs  for  the  period  from  1 March  1974  through 
1 March  1975  were  reviewed  manually  to  extract  the  CLC  CPU  time 
data.  Separate  run  logs  had  been  maintained  for  Missile  Site 
Radar  (MSR)  and  Perimeter  Acquisition  Radar  (PAR)  tests  by  month 
and  the  data  was  recorded  in  minutes  of  CPU  time.  The  data  was 
tabulated  by  month  for  MSR  and  PAR  and  totaled  for  each  month. 
Monthly  totals  were,  thereafter,  converted  from  minutes  to  hours 
per  month,  coded,  keypunched,  and  stored  on  disk  in  file  2 of  the 
SDA  Data  Tape  data  set. 

The  statement  type  was  the  same  for  all  modules  since  all  programs 
had  been  written  in  CENTRAN.  The  Bell  System  Technical  Journal  - 
Special  Supplement*  (1975)  , page  S57,  was  used  as  a reference  for 
ODtaining  the  statement  rate  based  on  a logical  statement  mix. 
Using  this  information  in  conjunction  with  the  graph  found  on  the 
same  page  (S57)  led  to  the  determination  of  a statement  rate  of 
25  microseconds.  The  information  was  then  written  up,  coded, 
keypunched,  and  added  to  file  2 of  the  SDA  Data  Tape  data  set. 

The  test  run  data  was  acquired  by  reviewing  a large  number  of 
progress  reports  from  several  areas  covering  the  period  of  time 
under  study.  The  data  was  tabulated  by  test  scenario,  with  a 
column  for  total  number  of  tests  run  and  a column  for  number  of 
tests  run  to  completion.  After  all  data  was  collected,  the  re- 
sulting statistics  were  calculated,  written  up,  coded,  keypunched, 
and  added  to  the  file  2 data  set. 

Several  program  libraries  containing  the  desired  MW,  PW,  and  sup- 
port programs  were  listed  indicating  PIDENT  name,  number  of  in- 
structions, and  language  used.  To  these  listings  was  added  the 
mode  of  construction  for  each  module.  The  data  was  formatted, 

*The  Bell  System  Technical  Journal  - Special  Supplement,  page  S57, 
1975,  American  telephone  and  Telegraph  Company. 


14 


keypunched,  and  loaded  on  file  3 of  the  SDA  Data  Tape  data  set. 
Figure  3 is  a sample  portion  of  a printed  PIDENT  data  set  listing. 

During  the  early  stages  of  the  SDA  study  it  became  apparent  that 
some  of  the  data,  because  of  its  uniqueness,  would  lend  itself  to 
an  automatic  analysis  procedure.  For  this  reason  the  decision 
was  made  to  undertake  two  types  of  analyses,  one  automatic  and 
the  other  a manual  process. 

The  design  of  the  program  used  to  identify  the  source  problem  re- 
ports written  during  the  T§I  phase  also  incorporated  the  initial 
building  of  the  SDA  data  base  and  a provision  for  executing  the 
automatic  analysis. 

The  matching  of  automatic  analysis  criteria  with  appropriate  error 
categories  presented  some  problems,  however.  Because  this  activi- 
ty took  place  at  the  beginning  of  the  study,  experience  in  match- 
ing error  categories  with  problems  had  not  yet  been  developed. 
Moreover,  the  explanations  of  many  of  the  major  error  categories, 
as  set  forth  in  Annex  1 of  the  Statement  of  Work,  were  causing 
some  confusion,  and  it  was  not  clear  that  major  error  categories 
and/or  subcategories  could  be  added  if  the  need  arose.  As  a re- 
sult there  existed  some  questions  concerning  the  validity  of  the 
study  team's  interpretation  of  certain  error  categories. 

The  SDA  Data  Base  Build  program  incorporated  within  its  design 
the  task  of  identifying  the  source  problem  reports  written  during 
the  T§I  phase  and  extracting  from  them  the  following  data  used  in 
establishing  the  initial  SDA  data  base  record. 

Source  Record  SDA  Record 


MPR  Number 

Date  Written 

PIDENT 

DESC 

SOLN 


Master  Problem  Number 

Date  of  Discovery 

Module  in  which  Error  Occurred 

Problem  Description 

Correction  Description 


The  rest  of  the  design  involved  the  automatic  analysis  function, 
wherein  the  remainder  of  the  SDA  error  data  (date  of  correction, 
phase  in  which  error  occurred,  type  of  correction,  error  classi- 
fication) would  be  acquired.  The  criteria  devised  for  the  auto- 
matic analysis  function  are  outlined  in  Tables  2 and  3. 

As  the  final  step  in  the  automatic  analysis,  the  program  scanned 
the  newly  formed  SDA  data  base  record  for  blank  fields.  If  a 
blank  field  was  found,  the  Build  program  looked  at  the  next  re- 
cord in  the  source  data  base.  If  that  record  had  the  same  basic 
problem  report  number,  the  Build  program  performed  an  analysis 
of  it.  If  that  analysis  supplied  data  on  all  the  SDA  fields. 
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CnSRECON 

001113 

CENTRAN 

convention al 

COSSTART 

00007  2 

CENT RAN 

CONVENTIONAL 

COSSUCSR 

0OOA96 

CENTRAN 

CONVENTIONAL 

COSTAR Fw 

0UU727 

CENTRAN 

CONVENTIONAL 

COST  AS  EU 

000006 

CENTRAN 

CONVENTIONAL 

COSTASKR 

000942 

CENTRAN 

CONVENTIONAL 

COSTVPGM 

001427 

CENTRAN 

conventional 

CCSXAPOG 

OOOlH>3 

CENTRAN 

CONVENTION AL 

COSZAPDG 

OOO003 

CENTRAN 

CONVENTION/! 

COSZCOMM 

001095 

CFNTRAN 

conventional 

COSZDSLT 

000034 

CENTRAN 

CONVENTIONAL 

COSZOUMP 

000191 

CENTRAN 

CONVENTION  AL 

COSZMAIN 

000757 

CENTRAN 

CONVENTIONAL 

COSZMCCil 

000643 

CENTRAN 

CONVENTIONAL 

C0SZMC02 

000  o4: 

CENTRAN 

CONVENTIONAL 

coszmmap 

000242 

CENTRAN 

CONVENTIONAL 

COSZPHSE 

000 1 06 

CENTRAN 

conventional 

COSZPLOS 

000217 

CENTRAN 

CONVENTIONAL 

COSZRLST 

0000; 9 

CENTRAN 

CONVENTIONAL 

COSZSVID 

U00J34 

CENTRAN 

CONVENTIONAL 

COSZUSRL 

CO0066 

CENTRAN 

CONVENTIONAL 

COSZXTCB 

00J114 

CENTRAN 

CONVENTIONAL 

COUCH DC M 

GO 0966 

CENTRAN 

STRUCTURED 

COUCHKCM 

OOU621 

CENTRAN 

structured 

COUCMPCM 

00l6bl 

CFNTRAN 

STRUCTURED 

COUCH PXX 

000424 

CFnTRAN 

STRUCTURED 

COUCTLVP 

OOOSbO 

CENTRAN 

STRUCTURED 

COUOIOXX 

000941 

CFNTRAN 

STRUCTURED 

COUDLTCH 

001 12K 

CENTRAN 

STRUCTURED 

COUDMPOM 

UU1727 

CENTRAN 

CONVENTION  AL 

COUJMPXX 

000370 

CENTRAN 

CONVENTIONAL 

CGUOPSDM 

000  319 

CFNTRAN 

CONVENTIONAL 

COUOSKVP 

002009 

CFNTRAN 

STRUCTURED 

COUEDTDH 

000894 

CFNT’AN 

CONVENTION AL 

COU5PCVP 

0ol413 

CENTRAN 

STRUCTURED 

COUEPLCM 

001148 

CENTRAN 

STRUCTURED 

COUFPOUT 

»>0O29o 

CENTRAN 

CONVENTIONAL 

COUFWRIT 

000710 

CENTRAN 

CONVENTIONAL 

CUULCPCM 

001196 

CENTRAN 

STRUCTURED 

COULOKOM 

OO2O0C 

CENTRAN 

CONVENTIONAL 

COUMESCH 

OuO 138 

CENTRAN 

STRUCTURED 

C0UNTER3 

000248 

CENTRAN 

conventional 

COUPCPCM 

00142c 

CENTRAN 

STRUCTURED 

Figure  3.  PIDENT  Listing 
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TABLE  2.  AUTOMATIC  ANALYSIS  CRITERIA 


Criteria 

Type  of 
Correction 

Phase 

Error 

Category 

10th  § 11th  characters  of 
PI DENT  name 

DN 

FD 

PD 

PS 

SD 

DOCUMENTATION 

DESIGN 

WW020 

FN 

PR 

PY 

SF 

DOCUMENTATION 

REQUIREMENTS 

WWOlO 

UM 

DOCUMENTATION 

REQUIREMENTS 

Q0020 

First  8 characters  of 
PI  DENT  name  and 

TYP  * P or  BLANK 
MWXSDC-- 
MWXMSIMP 
EMXSDC-- 
PWXSDC-- 
EPXSDC-- 

PATCH 

REQUIREMENTS 

KKOlO 

TYP  = S or  C 
MWXSDC-- 
MWXISIMP 
EMXSDC-- 
PWXSDC-- 
EPXSDC-- 

SOURCE 

REQUIREMENTS 

KKOlO 

TYP  = D 

MWXSDC-- 

MWXMSIMP 

EMXSDC-- 

PWXSDC-- 

EPXSDC-- 

DOCUMENTATION 

REQUIREMENTS 

WWOlO 

First  3 characters  of 
PI DENT  name  and 

TYP  = P or  BLANK 
PMG 
PML 
PTC 
PTL 
PWG 

PATCH 

REQUIREMENTS 

NN020 

TYP  = solution  type  for  source  problem  report 
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TABLE  2.  AUTOMATIC  ANALYSIS  CRITERIA  (Continued) 


Criteria 

Type  of 
Correction 

Phase 

Error 

Category 

TYP  = S or  C 
PMG 
PML 
PTC 
PTL 
PWG 

SOURCE 

REQUIREMENTS 

NN020 

TYP  = D 
PMG 
PML 
PTG 
PTL 
PWG 

DOCUMENTATION 

REQUIREMENTS 

WWOlO 

First  8 characters  of 
PI DENT  name  and 

TYP  = P or  BLANK 
MWGZCONS 
MWGSCONS 
MWGLACDS 
MWDDGCON 
MWFIRCON 
MWGSITEC 

PATCH 

REQUIREMENTS 

NN020 

TYP  =!  S or  C 
MWGZCONS 
MWGSCONS 
MWGLACDS 
MWDDGCON 
MWFIRCON 
MWGSITEC 

SOURCE 

REQUIREMENTS 

NN020 

TYP  = D 

MWGZCONS 

MWGSCONS 

MWGLACDS 

MWDDGCON 

MWFIRCON 

MWGSITEC 

DOCUMENTATION 

REQUIREMENTS 

WWOlO 

MWX §D  and 

TYP  = P or  BLANK 

PATCH 

REQUIREMENTS 

NN020 

TYP  = S or  C 

SOURCE 

REQUIREMENTS 

NN020 

TYP  = D 

DOCUMENTATION 

REQUIREMENTS 

WWOlO 

TYP  = solution  type  for  source  problem  report 


TABLE  2.  AUTOMATIC  ANALYSIS  CRITERIA  (Concluded) 


Criteria 

Type  of 
Correction 

Phase 

Error 

Category 

Character  string  of 
PREFACE  in  problem 
description 

DOCUMENTATION 

CODE 

QQ070 

CRB  Category  5 
Rejected  - Transient 

NONE 

NA 

SSOlO 

CRB  Category  5 
Rejected  - Duplicate 

NONE 

NA 

PP020 

TYP  = H or 

Process  Code  = HDW  or 
10th  § 11th  characters  of 
PI DENT  name 

HARDWARE 

NA 

WOOD 

Major  Status  = DEFERRED  or 
Minor  Status  = NOT  APPROVED 

NOT  FIXED  (no  further  ana 

lysis) 

TESTID  = 99/0003  or 
Date  source  not  BLANK  or 
Date  end  test  not  BLANK 

SOURCE  (no  further  analysis) 

TYP  = P or 
TYP  BLANK 

PATCH  (no  further  analysis) 

TYP  = S or 
TYP  = C 

SOURCE  (no  further  analysis) 

TYP  = D 

DOCUMENTATION  (no  further  analysis) 

TYP  = solution  type  for  source  problem  report 


TABLE  3.  DATE  OF  CORRECTION  CRITERIA* 


HIERARCHY  OF  CRITERIA 

SELECTION 

Correction 

1st  Choice 

2nd  Choice 

3rd  Choice 

4th  Choice 

5th  Choice 

PUP  ACT 

SITE  ACT 

PUP  SCH 

DATE 

DATE 

PATCH 

DATE 

DATE 

DATE 

CR  REC 

STAT 

DATE 

DATE 

DATE 

DATE 

SOURCE 

SOURCE 

END  TST 

CR  REC 

STAT 

DATE 

DATE 

DATE 

DOCUMENTATION 

DOCUMENT 

CR  REC 

STAT 

DATE 

DATE 

HARDWARE 

CR  REC 

STAT 

If  the  date  of  correction  selected  was  greater  than  10/1/75 
a default  date  of  10/1/75  was  used  instead 


that  record  was  substituted  for  the  previous  SDA  data  base  record. 
If  the  analysis  did  not  supply  the  SDA  fields,  the  Build  program 
looked  at  the  next  source  problem  report.  This  procedure  was  fol- 
lowed until  a source  report  either  furnished  all  of  the  necessary 
data  or  there  were  no  more  source  problem  reports  having  the  same 
basic  problem  report  number.  This  procedure  resulted  in  either 
an  SDA  data  base  record  possessing  all  of  the  necessary  data  or 
a record  with  one  or  more  blank  fields.  Those  records  containing 
blank  fields  were  set  aside  for  later  manual  analysis.  Execution 
of  the  SDA  Build  program  led  to  the  initial  generation  of  the  SDA 
data  base,  with  2060  records  directly  resulting  from  the  automatic 
analysis  process. 

At  this  point,  during  discussions  with  RADC  personnel  at  the  com- 
pletion of  the  automatic  analysis  effort,  it  became  clear  that  the 
interpretation  uncertainties  suspected  earlier  regarding  certain 
error  categories  were  real.  A random  check  of  the  automatically 
analyzed  data  revealed  that  results  were  not  as  good  as  anticipa- 
ted. One  of  the  trouble  areas  at  first,  and  throughout  the  manual 
analysis  phase,  involved  the  Preset  Data  Base  and  Global  Variable/ 
Compool  Definition  error  categories.  A new  approach  to  resolving 
the  problems  in  these  two  categories  had  to  be  devised,  and  sever- 
al subcategories  had  to  be  added  to  each  as  well.  The  category 
requiring  the  most  corrections  to  automatically  analyzed  data  was 
Requirements  Compliance.  Initially,  this  category  was  interpreted 
as  applying  to  documentation  as  well  as  to  software;  however,  after 
discussions  with  RADC  personnel,  it  was  used  only  where  the  soft- 
ware changed  because  it  did  not  meet  requirements. 

A new  category,  Design/Requirements  Logic  errors,  with  several 
subcategories,  had  to  be  defined  to  accommodate  the  errors  that 
had  originally  been  assigned  to  Requirements  Compliance.  Finally, 
many  of  the  classifications  made  in  the  Documentation  Errors  cate- 
gory had  to  be  changed  because  they  pertained  to  design  and  re- 
quirements documents  as  opposed  to  other  documentation. 

After  the  SDA  data  base  had  been  built  and  the  automatic  analyses 
run,  it  was  applied  against  a program  that  looked  for  one  or  more 
blank  fields  in  the  data  base  record.  When  a record  with  blank 
fields  was  detected  it  was  listed,  showing  which  data  was  present 
and  which  fields  had  no  data.  Figure  4 is  a sample  portion  of  an 
analysis  listing  which,  along  with  a listing  of  the  source  prob- 
lem reports,  was  used  for  the  manual  analysis  phase  of  the  study. 

Copies  of  the  analysis  listing,  the  source  problem  report  listing, 
and  Annex  1 of  the  Statement  of  Work  were  distributed,  with  in- 
structions, to  members  of  the  technical  staff.  The  purpose  of  the 
initial  pass  through  the  analysis  listing  was  to  pick  out  those 
problem  reports  having  blank  fields  that  were  obvious  and  simple 
to  fill  in,  and  to  gain  experience  in  assigning  error  categories. 
Subsequent  passes  through  the  analysis  listing  involved  increas- 
ingly complex  analyses. 
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As  problems  were  encountered  they  were  discussed  and  the  conclu- 
sions were  circulated  as  updates  to  either  the  set  of  instruc- 
tions or  to  the  set  of  error  categories.  One  of  the  first  prob- 
lems examined  was  the  problem  report  having  more  than  one  solution. 
If  the  solutions  were  all  of  the  same  type,  the  first  record  en- 
countered was  used.  If  the  solutions  were  of  varied  types,  the 
order  of  priority  was:  requirements  documentation,  design  docu- 
mentation, source  code,  and  patches. 

The  SDA  data  base  was  updated  each  week  using  the  previous  week's 
manual  analysis  findings. 

Upon  completion  of  the  manual  analysis,  a clean-up  and  review  of 
the  SDA  data  was  initiated.  The  clean-up  effort  consisted  of 
scanning  a copy  of  the  SDA  data  base  for  obvious  keypunching  er- 
rors that  were  not  spotted  during  the  updates,  and  obvious  erron- 
eous assignments  such  as  might  occur  if  the  phase  and  type  of  cor- 
rection were  transposed,  for  example.  The  review  involved  the 
listing  of  selected  major  error  categories  that  had  offered  parti- 
cular difficulty  during  the  manual  analysis  phase,  and  scanning 
the  error  category,  phase,  and  type  of  correction  assignments  for 
consistency. 

A final  update  to  the  SDA  data  base  was  made  following  the  com- 
pletion of  the  clean-up  and  review  activities. 

With  the  exception  of  the  Computational  and  Logic  Error  categories, 
the  descriptions  of  the  other  categories  did  not  seem  to  be  suffi- 
cient for  the  beginner.  After  reviewing  categories  with  the  cus- 
tomer and  gaining  actual  experience  in  assigning  error  categories, 
however,  a better  understanding  of  how  to  apply  categories  to  the 
problem  reports  naturally  developed.  Within  a short  time  it  was 
discovered  that  additional  major  and  minor  categories  were  needed, 
and,  despite  a reluctance  to  generate  new  major  error  categories, 
it  became  necessary  to  do  so  in  two  instances:  Hardware  errors 
and  Design/Requirements  Logic  errors. 

Although  the  definition  of  new  minor  error  categories  was  not  as 
significant  in  its  impact,  caution  was  exercised  to  hold  the  num- 
ber to  those  few  considered  essential.  Careful  attempts  were  al- 
ways made  to  fit  problems  into  the  categories  already  established. 
Only  when  a reasonable  fit  was  lacking  were  new  minor  categories 
defined. 

One  major  category  that  caused  little  difficulty  was  User-Requested 
Changes,  for  it  fit  well  into  the  problem  report  status  structure 
of  Deferred  Improvement.  For  this  reason,  almost  without  excep- 
tion, all  problem  reports  with  a status  of  Deferred  Improvement 
were  assigned  to  one  of  the  User-Requested  Changes  subcategories 
and  assigned  a phase  of  NA  (Not  Applicable).  The  phase  assignment 
of  NA  was  used  because  these  were  not  errors  in  the  generally  ap- 
plied sense,  and  thus  were  not  introduced  into  the  system.  In  the 
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typical  case  the  solution  for  the  problem  report  was  not  imple- 
mented, resulting  in  a type  of  correction  assignment  of  Not  Fixed. 
Occasionally  a software  change  was  made  requiring  the  type  of  cor- 
rection assignment  to  fit  the  type  of  change. 


The  definition  of  several  of  the  major  error  categories  occupied 
considerable  time  and  attention  and  it  is  worthwhile  to  describe 
these  cases.  The  two  major  categories  that  proved  most  difficult 
were  Preset  Data  Base  and  Global  Variable/Compool  Definition  er- 
rors. The  main  problem  was  adjusting  to  a new  concept  of  the  two 
types  of  data  sets  because  of  the  manner  in  which  they  were  used 
on  the  project  that  produced  the  source  data  for  the  study.  For 
purposes  of  the  study,  a Global  Data  Set  was  defined  to  be:  one 
used  by  more  than  one  routine  and/or  subroutine,  and  whose  data 
may  be  defined  at  requirements  or  design  time,  and  defined  and/or 
modified  during  execution  time.  A Preset  Data  Set  was  defined  as: 
one  used  by  only  one  routine  or  subroutine,  and  whose  data  may  be 
defined  and/or  initialized  by  some  external  source  prior  to  its 
utilization  by  the  host  routine.  The  data  in  the  data  set  could 
be  either  fixed  or  variable.  The  key  to  differentiating  between 
these  two  categories  appeared  to  be  how  the  data  set  was  used;  by 
one  routine  or  by  several  routines. 


An  initial  misunderstanding  regarding  the  Requirements  Compliance 
category  led  to  its  use  for  all  documentation  errors  to  require- 
ments and  design  documents.  As  a consequent  approach  to  correct- 
ing the  situation,  requirements  and  design  document  errors  were 
separated  from  other  documentation  errors  and  a new  error  cate- 
gory (Design/Requirements  Logic  errors)  was  established. 


Table  4 lists  the  new  error  categories  generated  during  the  SDA 
study. 
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Category 

FF040 


GGIIO 


KK020 

KK030 

KK040 


MM070 

MM080 


NN060 

QQ130 


RR030 


SSOlO 

SS020 


SS030 


TT060 

UU040 

WOOO 


WWOOO 

WWOlO 

WW020 

WW030 
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TABLE  4.  SUMMARY  OF  NEW  ERROR  CATEGORIES 
Definition 


System/system  incompatibility 

Routine  fails  to  maintain  integrity  of  interface  data 


VS  timeout  on  fetch  or  store 

Macro  definition  error 

Delete  unneeded  macro  definition 


Delete  unneeded  definitions 
Length  of  definition  incorrect 


Add  new  variables 


Comments  error 

(this  error  category  was  originally  listed  as  NN040) 


Delivered  capability  in  error 


Transient  error 

Error  the  analyst  cannot  identify  in  order  to 
categorize 

Error  that  was  fixed,  but  the  designer  did  not  know 
why  the  fix  worked 


Erroneous  input  entry 

Noting  the  existence  of  numerous  non-critical  errors 
Hardware  Errors 


Design/Requirements  Logic  Errors 
Requirements  documentation  errors 
Design  documentation  errors 

Typographical/editorial  error/cosmetic  change  to 
design  or  requirements  documentation 


Section  4 

STATISTICAL  ANALYSIS  OF  ERROR  DATA 


The  information  provided  in  this  section  is  not  intended  as  an 
in-depth  analysis  of  the  error  data  collected.  Its  purpose  is 
merely  to  show  some  basic  relationships  that  might  be  used  as 
points  of  departure  for  further  study  and  analysis. 

4.1  FREQUENCY  OF  OCCURRENCE  BY  ERROR  CATEGORY 

A brief  examination  of  the  bar  chart  illustrated  in  Figure  5 rea- 
dily indicates  that  the  Recurrent  Errors  category  (PP)  represents 
a significant  percentage  of  errors.  However,  when  the  category 
is  broken  down  into  its  component  subcategories  it  can  be  seen 
that  861  of  the  Recurrent  Errors  are  the  result  of  duplicate  prob- 
lem reports.  The  cross-hatched  section  of  the  bar  represents  the 
actual  recurring  errors. 

Another  category  that  is  somewhat  misleading  is  Documentation  Er- 
rors (QQ) , wherein  85%  of  the  errors  pertain  to  program  prefaces. 

In  almost  all  cases  the  prefaces  were  written,  but  they  were  not 
in  the  format  prescribed  by  the  project's  Policies,  Procedures, 
and  Standards  (PPS)  manual.  The  cross-hatched  area  on  the  bar 
represents  the  remainder  of  the  documentation  errors  --  with  the 
exception  of  errors  to  Design/Requirements  Logic,  which  were  as- 
signed a separate  error  category  (WW) . 

4.2  FREQUENCY  OF  OCCURRENCE  BY  PHASE 

The  NA  (Not  Applicable)  phase  is  somewhat  all-purpose  in  that  it 
reflects  those  errors  for  which  there  can  be  no  phase,  such  as 
the  categories  User-Requested  Changes  (LL) , Operator  Errors  (TT) , 
Questions  (UU) , and  the  subcategory  Duplicate  Problem  Report 
(PP020) ; and  those  errors  for  which  the  phase  cannot  be  deter- 
mined due  to  insufficient  information.  Unidentified  Errors  (SS). 

The  large  number  of  errors  appearing  in  the  NA  phase  (see  Figure  6) 
is  significant,  prompting  the  observation  that  many  of  the  problem 
descriptions  furnished  in  the  problem  reports  were  deficient  in 
the  description  of  the  problem  incurred  or  were  inadequate  in  the 
quality  of  the  description. 

4.3  NUMBER  OF  ERRORS  BY  MONTH 

The  hypothesis  under  which  the  statistics*  shown  in  Figure  7 were 
collected  proposed  that,  as  the  T§I  phase  progressed  from  begin- 


* Statistics  compiled  do  not  reflect  hardware  errors 
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Figure  5.  Frequency  of  Occurrence  by  Error  Category 


NUMBER  OF  ERRORS 


NUMBER  OF  ERRORS 


Total  Errors 
Corrected  Errors 
Errors  Not  Corrected 


Total  Errors  Corrected  - 3377 
Total  Errors  Not  Corrected  - 2069 
iTotal  Errors  - 5446 


MONTHS 


Figure  7.  Number  of  Errors  by  Month 


ning  to  end,  errors  would  more  likely  be  corrected  in  the  earlier 
months  and,  unless  essential,  would  tend  to  be  rejected  or  defer- 
red (not  corrected)  toward  the  finish  of  the  T§I  phase.  As  it 
turned  out,  this  was  not  the  case.  However,  it  is  apparent  that 
even  though  fewer  problem  reports  (errors)  were  being  written  to- 
ward the  end  of  the  T§I  phase,  and  fewer  errors  were  being  cor- 
rected, the  number  of  errors  not  corrected  dropped  off  only  slight- 
ly. A greater  percentage  of  errors  was  not  being  corrected  in  the 
latter  half  of  the  period,  tending  to  support  the  hypothesis,  but 
the  expected  crossing  of  the  two  curves  (corrected  and  not  correct- 
ed errors)  did  not  occur. 

Another  interesting  observation  is  that  in  the  total  figures  for 
corrected  errors  and  errors  not  corrected,  an  almost  60/40  rela- 
tionship existed.  Extending  the  relationship  reveals  that  approx- 
imately half  of  the  problem  reports  written  were  never  corrected. 


Section  5 


CONCLUSIONS 


5.1  PROBLEMS  ENCOUNTERED 

The  major  problem  encountered  during  the  SDA  study  was  in  acquir- 
ing a valid  and  workable  understanding  of  the  meaning  or  inter- 
pretation of  the  error  categories.  It  was  important  that  this 
information  be  well  understood  at  the  outset  to  enable  the  correct 
coordination  of  automatic  analysis  criteria  with  the  appropriate 
error  categories.  Not  investigating  the  meanings  of  the  error 
categories  in  more  depth  at  the  beginning  of  the  study  period  di- 
minished the  effectiveness  and  efficiency  of  the  automatic  anal- 
ysis process.  For  similar  studies  in  the  future,  it  is  recom- 
mended that  several  days  early  during  the  contract  period  be  de- 
voted to  study  and  clarification  of  the  error  categories.  It  is 
also  recommended  that  any  accompanying  documentation  containing 
the  definitions  and  error  categories  be  revised  and  expanded  with 
more  detailed  definitions  and  possible  examples.  The  proper  as- 
signment of  error  categories  is  a key  to  the  study  of  error  reli- 
ability modeling. 

Another  difficulty  involved  problem  reports  that  often  identified 
problems  of  a general  nature  that  necessitated  corrections  to 
more  than  one  module,  and  frequently  corrections  of  more  than  one 
type.  It  was  not  uncommon  for  problem  solutions  to  result  in  a 
correction  to  both  the  program  and  the  documentation.  When  this 
occurred,  all  parts  of  the  solution  were  identified  under  one 
problem  report  number  with  different  suffixes.  The  task  was  to 
pick  out  a single  error  and  type  of  correction,  along  with  the 
other  information  that  represented  the  problem. 

In  the  automatic  analysis  process,  either  the  first  record  of  a 
series  or  the  first  record  having  a complete  set  of  the  needed 
data  was  chosen.  For  manual  analysis  it  was  felt  that  errors  in 
documents  were  of  a higher  priority  than  those  in  programs,  and 
the  solution  selected  to  represent  the  error  was,  in  order:  re- 
quirements document,  design  document,  source  code  change,  patch 
change.  If  the  solutions  were  all  of  the  same  correction  type, 
however,  the  first  record  of  the  listing  was  used. 

Similar  problems  existed  where  there  were  multiple  solutions  to 
a problem  report  and  it  was  evident  from  the  different  suffix 
records  that  there  was  more  than  one  error  involved;  or  the  report 
identified  more  than  one  error  and  provided  more  than  one  solution. 
Under  the  problem  report  number  system  employed  to  document  errors, 
only  one  error  per  problem  report  could  be  identified;  thus,  in 
the  two  instances  given  above,  a choice  had  to  be  made  as  to  which 
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error  to  document.  The  priority  order  was  the  same  as  stated 
previously  for  the  manual  analysis  procedure:  requirements  doc- 
ument, design  document,  source  code,  patches. 

To  prevent  this  type  of  situation  from  becoming  a problem,  the 
problem  reporting  system  should  allow  only  one  error  to  be  iden- 
tified per  problem  report,  or  the  data  collection  and  recording 
methods  should  be  modified  so  that  each  error  is  uniquely  iden- 
tified. 

5.2  EVALUATION  OF  ACQUIRED  DATA 

Some  of  the  data  collected  during  the  SDA  study  was  not  as  valid 
as  had  been  hoped.  The  major  portion  was  collected  after  the 
fact,  and  much  of  the  source  information  needed  had  already  been 
disposed  of  or  had  never  been  present.  To  compensate,  certain 
assumptions  and  substitutions  had  to  be  made.  An  example  of  this 
is  the  way  in  which  the  date  of  correction  was  determined  (see 
Table  3) . In  many  cases  the  type  of  correction  was  also  deduced 
through  a series  of  assumptions.  For  example,  if  no  type  of  cor- 
rection --  such  as  a patch  number  or  a statement  in  the  comments 
section  to  indicate  a source  or  document  change  --  was  apparent, 
the  existence  of  patch  dates  in  the  record  had  to  be  checked.  If 
patch  dates  were  carried  in  the  record  it  was  assumed  the  correc- 
tion was  a patch;  if  no  patch  dates  were  present,  the  next  step 
was  to  search  for  source  or  documentation  dates.  If  none  of  these 
types  of  dates  were  available,  the  search  would  move  to  the  status 
category  of  the  problem  report.  When  the  status  was  Review,  De- 
ferred or  Rejected,  the  assumption  was  that  the  error  had  not  been 
corrected;  on  the  other  hand,  it  was  assumed  that  for  any  other 
status  of  the  problem  report,  the  type  of  correction  was  a patch. 

The  assignment  of  error  categories  was  not  as  consistent  or  as 
accurate  as  it  should  have  been,  for  several  reasons.  The  prob- 
lem and  solution  descriptions  in  some  of  the  problem  reports  were 
either  non-existent  or  so  lacking  in  detail  as  to  be  essentially 
meaningless.  In  those  cases  that  defied  reasonable  assumption, 
the  error  category  assignment  necessarily  became  Unidentified. 

This  is  one  important  area  in  which  configuration  management  con- 
trol could  have  been  exercised  to  require  adequate  descriptions. 

Other  factors  that  would  have  contributed  to  the  study  became 
apparent  during  the  course  of  the  contract.  For  example,  it  be- 
came clear  that  the  time  to  start  collecting  data  in  support  of 
error  analysis  and  error  reliability  modeling  studies  is  shortly 
after  the  commencement  of  the  project  from  which  the  data  is  to 
be  collected.  Also,  built  into  the  problem  reporting  and  change 
management  control  systems  of  the  host  project  should  be  those 
tools  necessary  to  collect  and  store,  on  a timely  basis,  all  the 
data  that  would  be  needed  to  meet  the  requirements  of  an  error 
analysis  study.  Along  with  this  should  exist  a configuration  man- 
agement control  and  monitoring  system  to  ensure  the  accuracy  and 
completeness  of  the  data  collected. 
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5.3  OBSERVATIONS 

The  SDA  study  could  be  the  first  of  a possible  series  of  similar 
studies  possessing  unique,  built-in  characteristics.  A number  of 
projects  already  in  progress  (PACS,  PAVE  PAWS),  about  to  begin  ji 

(SPACETRACK) , or  projected  for  the  future  (Cobra  Judy,  Pacific  \\ 

Barrier,  Space-Based  Radar)  share  an  unusual  research  environment,  l\ 

four  aspects  of  which  are  of  particular  interest:  common  applica-  || 

tions  skills  (Ballistic  Missile  Defense  skills) , commonality  of  || 

code,  common  systems  test  architecture,  and  experienced  personnel. 


By  maintaining  a cadre  of  experienced  personnel,  the  skills  and 
experience  acquired  on  previous  similar  projects  can  be  applied 
readily  to  other  projects  or  to  new  projects  as  they  come  into 
existence.  This  aspect  of  the  environment  has  already  been  ex- 
perienced as  personnel  from  the  project  on  which  this  study  is 
based  have  moved  on  to  the  PACS,  PAVE  PAWS,  and,  most  recently, 
SPACETRACK  projects. 

The  PACS  and  SPACETRACK  projects  utilize  the  PAR  Weapons  software 
portion  of  the  established  data  base  used  for  this  study.  Approx- 
imately 37%  of  the  PACS  code  was  changed.  Although  there  is  no 
transferability  of  code  to  PAVE  PAWS  from  the  data  base  on  which 
this  study  was  based,  the  techniques  and  application  of  technology 
are  very  similar. 

In  the  cases  of  the  PACS  and  SPACETRACK  projects,  the  presence  of 
a proven  test  bed  and  an  experienced  test  team  should  have  a sig- 
nificant positive  impact  on  the  time  necessary  for  software  instal- 
lation and  acceptance.  This  facet  of  the  environment  should  also 
have  considerable  influence  on  the  quality  and  reliability  of  the 
delivered  product. 

By  assigning  to  projects,  such  as  those  cited  above,  experienced 
personnel  who  possess  the  desired  applications  skills,  the  train- 
ing period  required  for  future  familiarization  would  be  minimized. 
Benefits  would  also  accrue  in  terms  of  reliability  improvement. 

The  environmental  aspects  mentioned  in  these  latter  paragraphs 
represent  a unique  opportunity  in  reliability  and  error  prediction 
modeling.  The  software  data  base  used  for  the  SDA  study,  plus 
the  PACS  and  PAVE  PAWS  projects,  can  provide  valuable  data  for 
use  in  the  development  of  models,  whereas  the  SPACETRACK  project 
can  provide  an  opportunity  to  test  the  prediction  reliability 
of  those  models  already  developed.  The  SPACETRACK  project  could 
also  provide  additional  data  for  further  model  development. 
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SDA  ERROR  CATEGORIES 
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Category 

ID 


AAOOO 

AAOlO 

AA020 

AA030 

AA040 

AA041 

AA050 

AA060 

AA070 

AA071 

AA072 

AA080 

AA090 

AAIOO 

AAllO 

AA120 


Category 

COMPUTATIONAL  ERRORS 

Total  mimber  of  entries  computed  incorrectly 
Physical  or  logical  entry  number  computed  incorrectly 
Index  computation  error 
Wrong  equation  or  convention  used 
Mathematical  modeling  problem 

Results  of  arithmetic  calculation  inaccurate/not  as 
expected 

Mixed  mode  arithmetic  error 
Time  calculation  error 
Time  conversion  error 
Time  truncation/rounding  error 
Sign  convention  error 
Units  conversion  error 
Vector  calculation  error 
Calculation  fails  to  converge 
Quantization/truncation  error 


BBOOO 

BBOlO 

BB020 

BB030 

BB040 

BB050 

BB060 

BB061 

BB062 

BB070 

BB080 

BB090 

BBIOO 

BBllO 

BB120 

BB130 

BB140 

BB150 

BB160 

BB170 

BB180 


LOGIC  ERRORS 

Limit  determination  error 

Wrong  logic  branch  taken 

Loop  exited  on  wrong  cycle 

Incomplete  processing 

Endless  loop  during  routine  operation 

Missing  logic  or  condition  test 

Index  not  checked 

Flag  or  specific  data  value  not  tested 
Incorrect  logic 
Sequence  of  activities  wrong 
Filtering  error 

Status  check/propagation  error 
Iteration  step  size  incorrectly  determined 
Logical  code  produced  wrong  results 
Logic  on  wrong  routine 

Physical  characteristics  of  problem  to  be  solved 
were  overlooked  or  misunderstood 
Logic  needlessly  complex 
Inefficient  logic 
Excessive  logic 

Storage  reference  error  (software  problem) 
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Category 

ID  Category 

CCOOO  INPUT/OUTPUT  ERRORS 

CCOlO  Missing  output 

CC020  Output  missing  data  entries  (PH  ■ code) 

CC030  Error  message  not  output 

CC040  Error  message  garbled 

CC050  Output  or  error  message  not  compatible  with  design 

documentation  (including  garbled  output) 

(PH  = code) 

CC060  Misleading  or  inaccurate  error  message  text 

(PH  = design) 

CC070  Output  format  error  (including  wrong  location) 

CC080  Duplicate  or  excessive  output 

CC090  Output  field  size  inadequate 

CClOO  Debug  output  problem  (relative  to  design  documen- 

tation) 

CClOl  Lack  of  debug  output 

CC102  Too  much  debug 

CCllO  Header  output  problem 

CC120  Output  tape  format  error 

CC130  Output  card  format  error 

CC140  Error  in  printer  control 

CC150  Line  count/page  eject  error 

CC160  Needed  output  not  provided  in  design 

CC161  Insufficient  output  options 

DDOOO  DATA  HANDLING  ERRORS 

DDOlO  Valid  input  data  improperly  set/used 

DD020  Data  written  in  or  read  from  wrong  disk  location 

DD030  Data  lost/not  stored 

DD040  Data,  index,  or  flag  not  set  or  set/initialized 

incorrectly 

DD041  Number  of  entries  set  incorrectly 
DD050  Data,  index,  or  flag  modified  or  updated  incorrectly 

DD051  Number  of  entries  updated  incorrectly 

DD060  Extraneous  entries  generated  (table,  array,  etc.) 

DD070  Bit  manipulation  error 

DD071  Error  using  bit  modifier 

DD080  Floating  point/integer  conversion  error 

DD090  Internal  variable  error  (definition  or  set/use) 

DDIOO  Data  packing/unpacking  error 

DDllO  Routine  looking  for  data  in  non-existent  record 

DD120  Bounds  violation 

DD130  Data  chaining  error 

DD140  Data  overflow  or  overflow  processing  error 

DD150  Read  error 

DD151  All  available  data  not  read 

(DATA  HANDLING  ERRORS  continued  on  following  page) 


Category 

ID 


>:  DD160 

\ DD170 

[ DD180 

DD190 

i . DD200 


Category 

DATA  HANDLING  ERRORS  (Continued) 
Long  literal  processing  error 
Sort  error 
Overlay  error 

Subscripting  convention  error 
Double  buffering  error 


EEOOO  OPERATING  SYSTEM/SYSTEM  SUPPORT  SOFTWARE  ERRORS 

EEOlO  Language  produces  erroneous  machine  code 

EE020  OS  missing  needed  capability 


FFOOO 

FFOlO 

FFOll 

FF020 

FF030 

FF040 


CONFIGURATION  ERRORS 
Compilation  error 
Segmentation  problem 
Illegal  instruction 
Unexplainable  program  halt 
System/system  incompatibility 


GGOOO 

GGOlO 

GG020 

GG030 

GG040 

GG050 

GG060 

GG070 

GG080 

GG090 

GGIOO 

GGllO 


ROUTINE/ROUTINE  INTERFACE  ERRORS 

Routine  passing  incorrect  amount  of  data  (insuffi- 
cient or  too  much) 

Routine  passing  wrong  parameters  or  units 
Routine  expecting  wrong  parameters 
Routine  fails  to  use  available  data 
Routine  sensitive  to  input  data  order 
Calling  sequence  or  routine/routine  initialization 
error 

Routines  communicating  through  wrong  data  block 
Routine  used  outs4,de  design  limitation 
Routine  won't  load  (routine  incompatibility) 
Routine  overflows  core  when  loaded 
Routine  fails  to  maintain  integrity  of  interface 
data 


HHOOO 

HHOlO 

HH020 

HH030 


ROUTINE/SYSTEM  SOFTWARE  INTERFACE  ERRORS 

OS  interface  error  (calling  sequence  or  initializa- 
tion) 

Routine  uses  existing  system  support  software  in- 
correctly 

Routine  uses  sense/jump  switch  improperly 
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Category 

ID 


IIOOO 

IlOlO 

II020 

II030 

II040 


JJOOO 

JJOlO 

JJ020 

JJ030 

JJ040 

JJ050 

JJ060 

JJ070 

JJ080 

JJ090 

JJlOO 


KKOOO 

KKOlO 

KKOll 

KK020 

KK030 

KK040 


LLOOO 

LLOlO 

LL020 

LL021 

LL022 

LL023 

LL024 

LL025 

LL030 

LL040 

LL050 

LL060 

LL070 

LL080 


Category 

TAPE  PROCESSING  INTERFACE  ERRORS 
Tape  unit  equipment  check  not  made 
Routine  fails  to  read  continuation  tape 
Routine  fails  to  unload  tape  after  completion 
Erroneous  input  tape  format 


USER  INTERFACE  ERRORS 

Operations  request  or  data  card/routine  incompati- 
bility 

Multiple  physical  card/logical  card  processing  error 

Input  data  interpreted  incorrectly  by  routine 

Valid  input  data  rejected  or  not  used  by  routine 

Input  data  rejected  but  used 

Input  data  read  but  not  used 

Illegal  input  data  accepted  and  processed 

Legal  input  data  processed  incorrectly 

Poor  design  in  operator  interface 

Inadequate  interrupt  and  restart  capability 


DATA  BASE  INTERFACE  ERRORS 

Routine/data  base  incompatibility 

Uncoordinated  use  of  data  elements  by  more  than  one 
user 

VS  timeout  on  fetch  or  store 

Macro  definition  error 

Delete  unneeded  macro  definition 


USER- REQUESTED  CHANGES 

Simplified  interface  and/or  convenience 

New  and/or  enhanced  functions 

CPU 

Disk 

Tape 

Input/Output 

Core 

Security 

New  hardware/OS  capability 

Instrumentation 

Capacity 

Data  base  management  and  integrity 
External  program  interface 


(PH  = NA) 
(PH  = NA) 
(PH  = NA) 
(PH  = NA) 
(PH  = NA) 
(PH  = NA) 
(PH  = NA) 
(PH  = NA) 
(PH  = NA) 
(PH  = NA) 
(PH  = NA) 
(PH  = NA) 
(PH  * NA) 
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Category 

ID 


i;  ! 

f'  : 

I 


t 


MMOOO 


MMOlO 

MM020 

MM030 

MM040 

MM041 

MM050 

MM060 

MM070 

MM080 


Category 

PRESET  DATA  BASE  ERRORS 

(a  data  set  that  is  generated  by  an  external  source) 
Data  or  operations  request  card  descriptions 
Error  message  text 

Nominal  default,  legal,  maximum/minimum  values 
Physical  constants  and  modeling  parameters 
Ephemeris  parameters  (short-lived  parameters  or 
interval  parameters) 

Dictionary  (bit  string)  parameters 
Missing  data  base  settings 
Delete  unneeded  definitions 
Length  of  definition  incorrect 


i 


NNOOO 

NNOlO 

NNOll 

NN020 

NN021 

NN030 

NN050 

NN060 


GLOBAL  VARIABLE/COMPOOL  DEFINITION  ERRORS 
Items  in  wrong  location  (wrong  data  block) 
Definition  sequence  error 
Data  definition  error 
Table  definition  incorrect 
Length  of  definition  incorrect 
Delete  unneeded  definitions 
Add  new  variables 


i 

• ' 

' 1 


PPOOO  RECURRENT  ERRORS 

PPOlO  Problem  report  reopened  or  previous  fix  in  error 

(TC  * none,  PH  = code) 

PP020  Problem  report  a duplicate  of  previous  report 

(reject  duplicates)  (TC  = none,  PH  = NA) 


QQOOO 


QQOlO 

QQ020 

QQ030 

QQ040 

QQ050 

QQ060 

QQ070 

QQ080 

QQ090 

QQIOO 

QQllO 

QQ120 

QQ130 


DOCUMENTATION  ERRORS 

(changes  to  documents  other  than  requirements  and 
design  specifications) 

Routine  limitation 
Operating  procedures 
Difference  between  flowchart  and  code 
Tape  format 

Data  card/operation  request  card  format 
Error  message 

Routine's  functional  description  (prefaces)  (TC  = 
documentation) 

Output  format 

Documentation  not  clear/not  complete 
Test  case  documentation 
Operating  system  documentation 
Typographical/editorial  error/cosmetic  change 
Comments  error 
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Jl 


m 


Category 

ID  Category 

RROOO  REQUIREMENTS  COMPLIANCE  ERRORS 

(code  was  changed  because  it  did  not  meet  the  re- 
quirements) 

RROlO  Excessive  run  time 

RR020  Required  capability  overlooked  or  not  delivered  at 

time  of  report 

RR030  Delivered  capability  in  error 

SSOOO  UNIDENTIFIED  ERRORS 

SSOlO  Transient  error  (TC  = none,  PH  = NA) 

SS020  Error  the  analyst  cannot  identify  in  order  to  cate- 

gorize (PH  = NA) 

SS030  Error  that  was  fixed,  but  designer  did  not  know  why 

the  fix  worked 


TTOOO  OPERATOR  ERRORS 

TTOlO  Test  execution  error 

TT020  Routine  compiled  against  wrong  Compool/Master  Common 

TT030  Wrong  data  base  used 

TT040  Wrong  master  configuration  used 

TT050  Wrong  tape(s)  used 

TT060  Erroneous  input  entry 

UUOOO  QUESTIONS 

(MPR  used  as  a vehicle  to  ask  a question  or  make  a 
statement) 

UUOlO  Data  base  (PH  = NA) 

UU020  Master  configuration  (PH  = NA) 

UU030  Routine  (PH  = NA) 

UU040  Noting  the  existence  of  numerous 

non-critical  errors  (PH  = NA) 


WOOO  HARDWARE  ERRORS 

(TC  = hardware,  PH  = NA) 


WWOOO  DESIGN/REQUIREMENTS  LOGIC  ERRORS 

WWOlO  Requirements  documentation  errors  (MPRs  written 

against  requirements  documentation) 

WW020  Design  documentation  errors  (MPRs  written  against 

design  documentation) 

WW030  Typographical/editorial  error/cosmetic  change  to 

design  or  requirements  documentation 


SI  Sybol 


Formuh 


I 


METUC  SYSTEM 


BASBUNTTS: 

Qiwntltv  _ UbJI 


length 

metre 

m 

meM 

kilogram 

kg 

time 

aecond 

a 

••t 

electric  ciurent 

ampere 

A 

themodynemic  temperature 

kelvin 

K 

amount  of  lubetance 

mole 

mol 

luminoua  intenaity 

candela 

cd 

SUmEMENTABY  UNITS: 

plane  angle 

radian 

rad 

(olid  an^ 

ateradian 

tr 

... 

OEUVED  UNITS: 

Acceleration 

metre  per  aecond  aquared 

m/a 

activity  (of  a radioactive  aource) 

diaintegratlon  per  aecond 

(diaintagrationVk 

angular  accalaration 

radian  per  aecond  aquared 

rad/a 

angular  velocity 

radian  par  aecond 

radia 

area 

aquare  OMtre 

m 

denaity 

kilogram  per  cubic  metre 

kg/m 

electric  capacitance 

farad 

F 

A-a/V 

electrical  conductance 

aiemena 

S 

AW 

electric  Raid  atrangth 

voh  par  metre 

... 

V/m 

electric  inductance 

henry 

H 

V.aiA 

electric  potential  difference 

volt 

V 

W/A 

electric  raaiatance 

ohm 

VIA 

electromotive  force 

volt 

V 

W/A 

energy 

joule 

1 

N>m 

entropy 

joule  per  kelvin 

... 

VK 

force 

newton 

N 

kg-m/a 

frequency 

hertz 

Hz 

(cydeVa 

illuminance 

lux 

lx 

Im/m 

luminance 

candela  per  aquare  metre 

... 

cd/n. 

luminoua  flux 

lumen 

im 

cdwr 

magnetic  field  atrength 

ampere  par  metre 

... 

A/m 

magnetic  flux 

weber 

Wb 

V« 

magnetic  flux  denaity 

teaia 

T 

Wbhn 

magitetomotive  force 

ampere 

A 

power 

watt 

W 

preaaure 

pMcal 

Pa 

KVm 

quantity  of  electricity 

coulomb 

C 

A*a 

quantity  of  heat 

joule 

1 

N-m 

radiant  intanaity 

watt  per  ateradian 

... 

W/ar 

apadfic  heat 

joule  per  kilogram-kelvin 

... 

VkrK 

atreaa 

paacal 

Pa 

Nmi 

thermal  conductivity 

watt  per  metre-kelvin 

... 

W/ni.K 

velocity 

metre  per  aecond 

... 

m/a 

viacoaity.  dyiMfflic 

paacal-aacond 

... 

Pa.a 

viacoaity,  kiaamatic 

aquare  metre  per  aecond 

... 

m/a 

vohaga 

vdt 

V 

W/A 

voluiiM 

cubic  matre 

... 

m 

wavenumber 

reciprocal  matre 

(wavej/m 

work 

joule 

i 

N-m 

j 


SIPREnXES: 


Multiplication  Factora 

Prefix 

.«U  Syn 

1 000  000  000  000  • to<> 

lera 

T 

1 000  000000  • to* 

8<S« 

t; 

1 000  000-  to* 

mega 

M 

i 000-  to* 

kilo 

k 

too  - 10* 

hedo* 

h 

10-  10' 

daka* 

da 

0.1  - 10-' 

deci* 

d 

0.01  - 10-* 

<»mi* 

c 

0.001  - 10-* 

mllli 

m 

0.000  001  - 10-* 

micro 

M 

0.000  000  001  - 10-* 

naim 

n 

0.000  000  000  001  - 10-'* 

if 

P 

0.000  000  000  000  001  - 10-" 

r 

0.000  000  000  000  000  001  - 10-'* 

atio 

a 

* To  ba  avoldod  whm  poMlble. 


J 


MISSION 

of 

Rome  Air  Develojyrnent  Center 


RADC  plems  and  conducts  research,  exploratory  and  advanced 
development  programs  in  ctxmmind,  control,  euid  conmunications 
(C^)  activities,  and  in  the  areas  of  information  sciences 
and  intelligence , The  principal  technical  mission  areas 
are  communications,  electromagnetic  guidance  and  control, 
surveillance  of  ground  and  aerospace  objects,  intalligmice 
data  collection  and  handling,  information  system  technology, 
ionospheric  propagation,  solid  state  sciences,  microwave 
physics  and  electronic  reliability , maintainability  and 
compatibility . 


